<html>
<head>
<title>OSPFv2 Testsuite - Documentation</title>
<link rel="stylesheet" href="testsuite.css" type="text/css">
</head>
<body>
<br><hr>
<div align="center">
<a href="http://ssfnet.d-jacob.net">Home</a> |
<a href="index.html">Up</a> |
<a href="testcases.html">Previous</a> |
Next
</div>
<hr><br>
<h1>Test Results</h1>

The execution of the test suite generates one of three possible verdicts for
each test case. <i>PASSED</i> is assigned, when the OSPF implementation behaves
exactly as required by the specification. <i>FAIL</i> is assigned, whenever an
offence against the specification is found. <i>INCONCLUSIVE</i> is used for test
cases, where no verdict can be assigned. This is the case for all tests cases
which test parts of the specification which are not yet implemented. The test cases, 
which only test features, that are not 
implemented yet are not considered in the following part of this section, since the
result is clear. Nevertheless, these tests have to be performed and evaluated once
the according features are implemented in OSPF. Table
3 shows an overview of test results for the current OSPF
implementation version 0.1.5.<p>

Some of the test cases are <i>INCONCLUSIVE</i> at
the moment. These test cases indicate, that they test both, features that are implemented,
but also features that are not implemented yet. The test steps for the features that were
implemented must have got a <i>PASS</i> verdict, since the complete result would have been
a <i>FAIL</i> otherwise. The <i>PASS</i> verdicts indicate, that the according features
have been implemented according to the RFC. The most interesting results are the
test cases, which have <i>FAIL</i> verdicts. These are now discussed in detail.<p>

<h3>Test 1.1 -- hello_addressing_and_timing</h3>

This test case verifies, if hello packets are sent periodically and addressed to
the multicast address <i>AllSPFRouters</i>. As discussed before, multicast is 
not yet available. OSPF
packets are always unicast in the current OSPF version, which causes this test
to fail.<p>

<h2>Earlier Results</h2>

As mentioned before, the test suite has two main purposes. It can be used for
conformance testing and for regression testing as well. Therefore, the tests are
executed, everytimes there is a new version of OSPF. Besides of the results
presented above, the test suite helped uncovering a number of bugs during the
development of OSPF. Because the test suite covers the most parts of the RFC,
performing the tests guarantees, that wide parts of the actual code are exected
at some time. Especially the handling of exceptions or code dealing with special
configurations can be tested in this way. Indeed, the execution of the test
cases helped finding some unhandled exceptions. Mostly these were "standard"
bugs, like unhandled <tt>NullPointerExceptions</tt>, 
<tt>ConcurrentModificationExceptions</tt> or casts to wrong classes, which could
be fixed easily. This helped to make the code more stable. Surprisingly only a
small number of errors was related to misunderstanding the RFC.<p>

One error for example, caused most of the Flooding tests to fail. When an LSA
has to be flooded, it is set on the retransmission list. Then the a flood timer
is started, which sends all LSAs from this list, when it fires. Then this timer
must be set again, since SSFNet timers are single shot timers. Again, a
"standard" bug caused the whole flooding procedure to fail. When the
retransmission list is empty, the timer is not restarted. But the timer object
exists further on. When the timer had now to be restarted, it was checked for a
<tt>null</tt> value, instead of testing, whether the timer is cancelled. Because
of this, the flooding timer was never restarted and no LSAs are flooded at all.
After this bug was fixed, most flooding tests were passed.<p>

These experiences show, how useful extensive testing is for the development
process. Using the appropriate tests helps finding errors early. Detailed test
cases help restricting the possible sources of errors to well defined regions of
the source code.<p>

<h2>Complete test results</h2>
<div align="center">
<table border=1>
<tr bgcolor="#CCCCCC"><td><b>Test. No.</b></td>
<td><b>Verdict</b></td>
<td><b>Problems / Remarks</b></td></tr>
<tr bgcolor="#CCCCCC"><td colspan=3><b>Hello Protocol Tests</b></td></tr>
<tr><td>
1.1  </td><td> FAIL </td><td> missing multicast support</td></tr>
<tr><td>
1.10 </td><td> PASS </td><td> </td></tr>
<tr><td>
1.11 </td><td> PASS </td><td> automatic address assignment</td></tr>
<tr><td>
1.12 </td><td> PASS </td><td> </td></tr>
<tr bgcolor="#CCCCCC"><td colspan=3><b>Flooding and Adjacency Tests</b></td></tr>
<tr><td>
2.2  </td><td> INCONCLUSIVE </td><td> virtual links not supported</td></tr>
<tr><td>
2.3  </td><td> PASS </td><td>  </td></tr>
<tr><td>
2.4  </td><td> PASS </td><td>  </td></tr>
<tr><td>
2.5  </td><td> PASS </td><td>  </td></tr>
<tr><td>
2.6  </td><td> PASS </td><td>  </td></tr>
<tr><td>
2.7  </td><td> PASS </td><td>  </td></tr>
<tr><td>
2.8  </td><td> PASS </td><td>  </td></tr>
<tr><td>
2.9  </td><td> INCONCLUSIVE </td><td> AS-external LSAs not yet supported </td></tr>
<tr><td>
2.10 </td><td> PASS </td><td>  </td></tr>
<tr><td> 
2.16 </td><td> PASS </td><td>  </td></tr>
<tr><td> 
2.17 </td><td> PASS </td><td>  </td></tr>
<tr><td> 
2.18 </td><td> PASS </td><td>  </td></tr>
<tr><td> 
2.19 </td><td> PASS </td><td>  </td></tr>
<tr><td> 
2.20 </td><td> PASS </td><td>  </td></tr>
<tr><td> 
2.21 </td><td> PASS </td><td>  </td></tr>
<tr bgcolor="#CCCCCC"><td colspan=3><b>LSA Tests</b></td></tr>
<tr><td>
3.22 </td><td> PASS </td><td>  </td></tr>
<tr bgcolor="#CCCCCC"><td colspan=3><b>Configuration and Formatting Tests</b></td></tr>
<tr><td> 
5.1  </td><td> PASS </td><td> implicit </td></tr>
<tr><td>
5.2  </td><td> PASS </td><td> implicit </td></tr>
<tr><td>
5.9  </td><td> PASS </td><td> implicit </td></tr>
<tr><td>
5.10 </td><td> PASS </td><td> implicit </td></tr>
<tr><td>
5.11 </td><td> PASS </td><td> implicit </td></tr>
<tr><td>
5.12 </td><td> PASS </td><td> implicit </td></tr>
<tr><td> 
5.14 </td><td> PASS </td><td>  </td></tr>
<tr bgcolor="#CCCCCC"><td colspan=3><b>Additional Tests</b></td></tr>
<tr><td> 
6.1 </td><td> PASS </td><td>  </td></tr>
<tr><td>
6.2 </td><td> PASS </td><td>  </td></tr>
<tr><td>
6.3 </td><td> PASS </td><td>  </td></tr>
<tr><td>
6.4 </td><td> PASS </td><td>  </td></tr>
</table>
</div><br>
<b>Table 5</b><p>

<br><hr>
<div align="center">
<a href="http://ssfnet.d-jacob.net">Home</a> |
<a href="index.html">Up</a> |
<a href="testcases.html">Previous</a> |
Next
</div>
<hr><br>

<i>Last update: 2002-11-11, Dirk Jacob (<a
href="mailto:dirk@d-jacob.net">dirk@d-jacob.net</a>)</i>
</body>
</html>
